class: center, middle, inverse, title-slide .title[ # Multiple Linear Regression ] .subtitle[ ## Lecture 3 ] .author[ ### Manuel Villarreal ] .date[ ### 08/28/24 ] --- ### Recap - We wanted to study the "**linear**" association between age and participant's blood pressure. -- - We implemented a simple linear regression of the form: `$$\text{blood pressure}_i = \beta_0 + \beta_1 \text{age} + \epsilon_i$$` -- - And we assumed that `\(\epsilon_i \overset{iid}{\sim} N(0, \sigma^2)\)`. -- - However, the residual `\((\hat{\epsilon}_i)\)` plots showed that a participant's sex could be confounding the effects of age on blood pressure. -- - In order to be able to make valid inferences first we need to **stratify** our model to be able to account for the effect of sex on blood pressure. --- ### Multiple linear regression - A multiple linear regression model is an extension of the typical approach were we add more than on predictor. -- - For example, we might be interested in how **height** changes as a function of age and socioeconomic status. -- - Rent price of an apartment depending on how old it is, the number of rooms it has, the squared footage, and the area it is located in. -- - **Note** that the predictors that we use can be continuous or discrete. Mathematically this will make no difference, however, our interpretation of the inferences we make will depend on the variable's level. --- ### Multiple linear regression We can expressed the multiple linear regression as: `$$Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \cdots + \beta_p X_{ip} + \epsilon_i$$` -- - Where `\(X_{ip}\)` is the value of the `\(p-th\)` independent variable for the `\(i-th\)` participant. -- - We can write the same equation in a different form: `$$Y_i = \sum_{j = 0}^p \beta_j X_{ij} + \epsilon_i$$` -- - Notice that we still have a single error term. This means that we can keep our assumptions about the errors from the simple linear regression model. --- ### Additional assumptions - Now that we have multiple predictors in our model we have to add an extra assumption about those predictors. -- - The predictors or covariates `\(X_{ij}\)` for `\(j = 1, \dots, p\)` are linearly independent. -- - What does this mean? Well, that we can't express one of our predictors as a linear function (like the multiple linear regression equation) of other predictors! -- - When we have two or more independent variables that are not linearly independent then we say that the variables are collinear. --- ### Collinearity - When the linear independence between predictors is not met, we will have inaccurate estimates and inaccurate intervals for our parameters. -- - Additionally, if the problem is bad enough we might not be able to obtain the OLS estimators of our parameters. -- - One way that we might use to check for collinearity problems is by calculating the correlation between the predictors in the model. -- ``` r r_age_height <- cor(x = blood$age, y = blood$height) ``` -- - Notice that we can't get the correlation between a categorical variable like sex and a numeric variable. --- ### Interpreting the coefficients - When we had a single independent variable in our model the interpretation of the coefficient was straightforward. However, when we add a second independent variable the way in which we interpret those values has to change. -- - In general, for a linear model of the form: `$$Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \cdots + \beta_p X_{ip} + \epsilon_i$$` we say that: - `\(\beta_j\)` is the change in the expected value of `\(Y_i\)` for a one unit increase in the variable `\(X_j\)` while holding all other variables constant. -- - This last part (holding all other variables constant) means that the value of `\(\beta_j\)` can only be interpreted as the change in expectation for participants who are similar on other variables except for a one unit difference in `\(X_j\)`. --- ### Interpreting the coefficients - When we have a mixture of both continuous and discrete independent variables something interesting happens. -- - From our example, let's say that we add the **sex** variable to our model that already includes age. `$$\text{blood pressure}_i = \beta_0 + \beta_1\text{age} + \beta_2 \text{sex} + \epsilon_i$$` -- - Given that **sex** can only take two values in our data base, R will interpret one of those values as a 1 and the other as a 0. Generating what we refer to as a **base-line** group. -- - For example, if female is assigned the value of 0 then `\(\beta_0\)` can be interpreted as the expected blood pressure of a female who is 0 years old --- ### Interpreting the coefficients - On the other hand, `\(\beta_0 + \beta_2\)` can be interpreted as the expected blood pressure of a male that is 0 years old. This means that `\(\beta_2\)` is the difference between a male and a female participant who are 0 years old. -- - In other words, because we added an independent variable that can only take two variables, we can visualize our multiple linear regression as two different lines, one for each value of our discrete independent variable. -- - If we where to add a third discrete independent variable, then the **base-line** group would be the combination of the two levels of our discrete variables that are assigned the value of 0. -- - For example, if we added **socioeconomic status** as a predictor with 3 levels, low, medium and high, then `\(\beta_0\)` would be interpreted as the expected blood pressure of a female participant in the low socioeconomic status that is 0 years old. --- ### Ordinary Least Squares estimators - In comparison to the simple linear regression approach, finding the OLS estimators is more difficult, and requires understanding of linear algebra notation. -- - Therefore, instead of calculating the estimates by solving the normal equations we will use a solver already programmed in R. -- - For a linear regression model we use the function `lm()`. --- ### Example SLR with `lm()` - Going back to our example, we can compare the values obtained using the `lm()` function with our estimate of the simple linear regression: -- ``` r model_bp_age <- lm(formula = blood_pressure ~ age, data = blood) ``` -- the estimated intercept using this function was `\(\hat{\beta}_0 =\)` 104.15 and the estimate of the slope associated to age was `\(\hat{\beta}_1 =\)` 0.5. -- - When saved into an r object, the `lm()` function will return multiple values that can be useful, like the estimated coefficients (the OLS) and the model residuals. --- ### Output of the `lm()` function - The output of the function will be saved in the form of a list. A list is an R object that can store multiple different objects like matrices, arrays, vectors and scalars. -- - To access a particular stored variable we can use the \$ sign after the name of the object and then the name of the variable we want to look at. -- - For example, to access the estimated values of the parameters in the model we can do call the `coefficients` variable stored in the list. If I saved the output from the `lm()` function on an R object named `model_bp_age`, then: -- ``` r model_bp_age$coefficients ``` ``` ## (Intercept) age ## 104.1503322 0.5047537 ``` prints the values of the OLS estimators of each parameter. --- ### Output of the `lm()` function - This values will be stored in a vector format and therefore we can access to each of them individually by using square brackets after "`coefficients`" and then the index of the value we want to use. -- - For example, `model_bp_age$coefficients[1]` will return the value of the intercept, while `model_bp_age$coefficients[2]` will return the value of the slope associated to `age`. --- ### Residuals - The residuals are also stored and organized in a vector: ``` r model_bp_age$residuals ``` -- <div class="datatables html-widget html-fill-item" id="htmlwidget-72a744e1a357b3e52df6" style="width:100%;height:auto;"></div> <script type="application/json" data-for="htmlwidget-72a744e1a357b3e52df6">{"x":{"filter":"none","vertical":false,"data":[[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200],[1.217,-2.098,1.826,3.131,4.626,-1.564,2.912,0.555,0.341,-0.24,6.831,-1.974,0.131,0.021,3.817,5.636,2.979,1.264,9.602,-3.474,2.307,1.231,-1.479,7.326,-1.869,4.055,5.702,0.407,6.507,6.621,2.341,-3.259,5.112,-0.645,-1.588,-2.126,2.517,0.736,0.226,-0.779,-0.955,-1.979,-0.769,-2.936,-2.383,5.541,-1.907,3.007,0.217,1.45,-0.379,-6.069,6.426,5.802,0.226,1.545,-1.893,4.021,-4.588,5.526,1.631,5.474,5.85,1.721,0.317,2.917,3.302,6.174,-1.074,5.912,3.631,3.112,1.312,1.512,-2.593,4.131,-0.388,3.421,5.593,5.336,4.269,0.85,3.964,2.902,0.321,-0.155,2.788,4.731,5.217,-0.479,3.764,6.293,-0.864,3.312,-1.069,-3.779,-0.974,-1.593,5.431,3.098,-3.174,-1.269,2.545,-2.612,-5.407,-2.598,3.841,-5.159,-5.102,-6.074,-2.812,1.445,0.231,-2.093,0.612,-0.45,-1.393,-4.098,-1.288,0.855,0.341,-2.869,-4.869,-1.936,-5.359,-4.621,-4.174,1.741,-0.383,-5.383,-6.088,-1.574,2.383,-2.559,4.607,-2.317,-1.279,-6.369,-4.293,-3.812,-2.879,3.502,0.021,-3.874,0.312,-2.017,1.936,0.983,3.445,-1.412,3.845,0.512,-4.669,-1.964,-4.879,-5.979,-7.302,-2.007,-1.693,-1.007,-1.293,0.702,1.407,-3.45,10.16,0.8169999999999999,-4.579,-5.979,-0.483,-1.607,-3.888,-0.593,-7.193,-3.693,2.331,-1.798,-1.883,-0.274,3.431,-5.679,-1.874,-1.093,-4.212,-0.512,-5.683,-10.617,-6.393,-1.788,-1.688,1.421,-3.707,-5.264,2.036,-4.607,0.245,1.936,-0.779,2.021,-2.064,-0.017]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th>id<\/th>\n <th>residuals<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"pageLength":3,"columnDefs":[{"className":"dt-right","targets":[0,1]},{"name":"id","targets":0},{"name":"residuals","targets":1}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[3,10,25,50,100]}},"evals":[],"jsHooks":[]}</script> --- ### Output of the `lm()` function - Other useful value returned by the `lm()` function are the "fitted values" which are stored with the name `fitted`. These are the values of `\(\hat{\mu}_i\)`. -- - We can also use other R functions like the `summary()` and `anova()` functions that take advantage of the output to do statistical significance tests . --- ### Model notation - The first argument of the `lm()` function will be the model's **formula**. -- - This argument can be entered two ways, by writing the name of a data frame followed by the dollar sign and the name of the variable we want to use, for example: ``` r lm(formula = blood$blood_pressure ~ blood$age) ``` -- - The second way is by using the **data** argument of the `lm()` function, this way we can just write the name of the variables of the data frame in the formula section and then the name of the data frame for the data argument. ``` r lm(formula = blood_pressure ~ age, data = blood) ``` --- ### Multiple linear regression - Like we mentioned at the start, a multiple linear regression is a linear model where we have 2 or more independent variables and a single dependent variable. -- - For example, from our example using the blood pressure data, we noticed that the sex variable was associated with our independent variable. -- - Formally, we can express this new model as: `$$\text{blodd pressure}_i = \beta_0 + \beta_1\text{age}_i + \beta_2\text{sex} + \epsilon_i$$` -- - We can estimate the parameters ( `\(\beta_0\)`, `\(\beta_1\)`, and `\(\beta_2\)`) of this new model using the `lm()` function: ``` r bp_age_sex <- lm(formula = blood_pressure ~ age + sex, data = blood) ``` --- class: middle, center, inverse ### Now it's your turn!